Catalyst is a PyTorch framework developed with the intent of advancing research and development in the domain of deep learning. It enables code reusability, reproducibility and rapid experimentation so that users can conveniently create deep learning models and pipelines without writing another training loop. Catalyst framework is part of the PyTorch ecosystem – a collection of numerous tools and libraries for AI development. It is also a part of the Catalyst Ecosystem – an MLOps ecosystem that expedites training, analysis and deployment of deep learning experiments through Catalyst, Alchemy and Reaction frameworks respectively. We have used the well-known MNIST dataset having 10 output classes (for classifying images of handwritten digits from 0 to 9).

We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept. As a result, only $k$ rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction ($k$ divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1-4% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at https://github.com/lancopku/meProp

For the 0.3.3 release of PlaidML, support for running deep learning networks on macOS has improved with the ability to use Apple's native Metal API. Metal offers "near-direct access to the graphics processing unit (GPU)", allowing machine learning tasks to run faster on any Mac where Metal is supported. As previously announced, Mac users could accelerate their PlaidML workloads by using the OpenCL backend. In our internal testing, in some cases, we see an up to 5x speed up by using Metal over OpenCL. Next, run plaidml-setup to select the desired Metal-based device.